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ABSTRACT 

Several sampling procedures for collecting 
observational data on student activities were studied in an effort to 
determine their relative efficiency. The setting was a fifth grade 
Individually Prescribed Instruction (1P1) mathematics class of 
thirty-three pupils. A criterion measure was obtained by cumulating 
the measurements obtained on the thirt y- three pupils every thirty 
seconds for thirty minutes over a period of twenty days. Various 
observatic na3 designs were created by manipulating the following four 
va r iables : (i) the length of the observation period; (2) the time 

interval between recording observations during tne observation 
period; (3) the number of days (replications) of observing; and (4) 
the number of pupils observed each day. Through this procedure it was 
possible tc build observational design*, and compare results using 
eacn design with the criterion measure, thus determining the 
efficiency of the several sampling plans. Comprehensive tables are 
included showing Scott coefficients of agreement with the criterion 
of the various designs. (Author/LR) 
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AN INVESTIGATION OF THE EFFICIENCY OF 
VARIOUS OBSERVATIONAL PR OCEDURES 

by 

Mary Ann Kissel and John I/. Yeager 



Frequently an educational researcher must collect data about a 
class or group of pupils by way of direct observation of student activities 
in the classroom. Such direct observation procedures are time consuming 
and costly. The purpose of this study was to examine various observational 
procedures and sampling plans in order to determine which was the most 
efficient method for collecting this type of observational data. 

The overall plan of the research was first to obtain a criterion 
measure and then to compare the results of various observation procedures 
and sampling plans to this treasure* By manipulating several design variables, 
it was possible to build different observational designs and to test their 
efficiency by comparing the results obtained by them to the criterion* 

Procedure 

The setting of the investigation was a fifth grade Individually 
Prescribed Instruction (IPl) mathematics class consisting of thirty-three 
pupils. IPI is a program of individualized study, developed by the Learning 
Research and Development Center at the University of Pittsburgh, that 
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permits each pupil to progress through a given set of instructional objectives 
at a rate commensurate with his own unique abilities. Each pupil in the 
class was assigned randomly, on a daily basis, to one of four observers 
w ho recorded the activities that pupils engaged in during this period. The 
activity in which each of the pupils was engaged was recorded every thirty 
seconds for thirty minutes over twenty class periods. The criterion measure 
was the cumulative measure of the thirty-three pupils for the twenty 
class periods. 

Activity frequencies for each of the pupils were recorded on 
a revised form of the student observational form developed by Lindvall, 
Yeager, Wang, and Wood (1967) (see page 26), The major pupil activity 
categories were: (1) independent activity using materials; (2) independent 
activity not involving the use of materials; (3) activities involving two or 
more pupils; (4) activities involving one pupil and the teacher; (5) activities 
involving the pupil and a teacher aide; and, (6) teacher directed group 
instruction. Under each of the main categories were listed specific 
item9 of pupil behavior that were utilized by the observers when the 
activities of the pupils were recorded. At the beginning of each thirty 
second interval, the activity in which each pupil wai engaging was noted 
by the observer and recorded opposite one of the specific activity items. 
Before the data were analyzed, however, all pupil activity frequencies 
were cumulated according to the six major categories. 

Different observational designs were created by studying the 
four variables: (1) the length of the observation period; (2) the time 
interval between recording observations during the observation period; 

(3) the number of days (replications) of observing; and, (4) the number 
of pupils observed each day. The variables were chosen on the premise 
that they would interact to either increase or decrease the number of 
classroom observations required to reach agreement efficiently with 
the criterion measure. It was expected that if one or two of the variables 
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were held constant, the other variable would interact. Thus, if the time 
interval and the observation period were held constant, the number of days 
necessary for observation would be dependent upon the number of pupils 
observed. It was anticipated that in studying the variables in this way, 
the optimal observational procedure for each of the variables (or combination 
of variables) could bo established. Nevertheless, it was recognized that 
this study was constrained by several factors such as: the classroom and 
the grade level observed; the academic subject area; the size of the class; 
the observation variables and the design of the observation form; and, the 
observer training. 

The pool of criterion measures served as "population" data 
for this study. Sub-sets of data based on different observational procedures 
were computed to determine the degree of agreement between the results 
obtained from the different procedures and the criterion. The criterion 
measure was obtained by summing, over a period of twenty days, the 
total number of frequencies recorded every thirty seconds in each of 
the six major pupil activity categories for all thirty-three pupils. Scott 
Coefficients (1955) were computed to determine the degree of agreement 
between each set of measurements and the criterion. For this study, 
efficiency of an observation procedure was defined as the amount of time 
needed for the procedure to produce a . 35 level of agreement with the 
criterion. This acceptance level of agreement is in accordance with the 
standard set by Flanders (1967) in training observers to use the Flanders' 
Interaction procedures. However, rather than requiring that all indices 
be consistently ,85 or above, the general rule applied, for defining an 
acceptable number of days or pupils for this study, was that after the 
initial . 85 value had been obtained no three consecutive indices fell below 
a . 80 value. 

The observers for this study weve trained in ten one hour sessions 
prior to the start of the study. Only one training session was held each 
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day# The first three days were used for describing the study and its 
purpose and familiarizing the observers with the observation form and 
the IPI procedures# During these three days the observers were drilled on 
the items of the observation form and classroom simulation was provided 
through the use of video tapes. These three initial sessions were followed by 
six days of actual classroom recording# Discussions were held both before 
and after the training sessions to ensure uniform and accurate recording 
of pupil activities. Also, during this period inter -observe r reliability 
was checked. On the last day of the training period, all observers 
recorded the activities of the same eight pupils every thirty seconds for 
thirty minutes. The results indicated that an acceptable level of agree- 
ment had been reached among the observers. 

The first variable examined was the time interval or the time 
allotted between recordings. At the beginning of each time interval, 
a recording was made of the activity in which each pupil was engaged 
Different time intervals were investigated and the number of pupils and 
days required for each variation to attain ar d then consistently maintain 
an acceptable level of agreement with the criterion were determined. As 
previously stated, the criterion measure for this study was a twenty day 
cumulative measure of all pupils. 

The smallest time period studied was the thirty second interval. 
Table 1 shows the Scott Coefficients that resulted when the thirty second 
interval for one through thirty-three pupils extending one through twenty 
days was compared with the criterion measure. The table represents 
cumulated days going down the columns and cumulated pupils going across 
the rows. For example, . 17 is the index of agreement between the 
cumulative activities of two pupils recorded every thirty seconds for thirty 
minutes for one day and the criterion measure. 

Examination of the matrix shown in Table 1 shows that when 
one pupil was selected randomly each day and his activities were recorded 




4 



5 



every thirty seconds for thirty minutes, it took twelve days to reach a 
.85 level of agreement. Further inspection of column one also shows 
that the level of agreement, with some variability, reached .91 at the end 
of twenty days. Increasing the number of pupils observed from one to 
two produced only slight changes in the indices obtained; but when three 
pupils were chosen each day, only eight days of observation were required 
to reach a consistent . 85 or higher level of agre mcnt. 

It is interesting to note that even though the classroom was 
individualized and the recording of the activities of only one pupil each 
day produced an acceptable level after twelve days, the activities recorded 
for thirty-three pupils on one day did not yield an acceptable level of 
agreement. However, when the activities of fourteen pupils were recorded 
for only two days, a .85 level was attained and when the student number 
was increased to sixteen, an even more consistent acceptable pattern 
resulted. Furthermore, when the student number was only twelve, three 
days of observing were necessary to reach the . 85 level of agreement and, 
except for a few slight reversals to indices below . 85 , this level was either 
maintained or raised both down and across the remaining right side of the 
matrix. When the matrix was triangularly divided from the bottom left 
to the top right, it could be easily seen that as the number of observation 
days increased, the number of pupils that it wa3 necessary to observe 
decreased or as the number of observed pupils increased, the number of 
necessary observation days decreased. It was also noted that the lower 
half of the triangle contained coefficients of . 90 or above except for a few 
in the low eighties near the upper dividing line of the triangle, the reby, 
confirming the conclusion that the number of days or pupils could be 
decreased with acceptable levels still being attained. 

The second time interval compared to the twenty day criterion 
was the every one minute interval. The coefficients computed for this 
interval are shown in Table 2. Although there appeared to be a close 
overall similarity between these levels of agreement and those of the every 



thirty seconds interval, there were 'several differences noted. When 
only one pupil’s activities were recorded each day, fourteen days were 
required to reach a consistent acceptable level of agreement rather than 
twelve. Also, there were nine acceptable coefficients produced on the 
first day of recording; there were none when the smaller recording 
interval of every thirty seconds was used. 

Further comparison of Tables 1 and 2 revealed that although the 
recordings of every thirty seconds and every one minute for three pupils 
differed only slightly, the activities of five pupils rather than three were 
required to produce a consistent level of . 85 or above in eight days. 
Nevertheless, the number of acceptable coefficients in the lower triangular 
matrix were approximately the same as those found in the every thirty second 
matrix. Therefore, the general conclusion was that although much more 
information was obtained of the activities of the pupils by recording obser- 
vations every thirty seconds, the one minute interval was satisfactory if 
a sufficient number of pupils were observed. 

The next interval examined with respect to the twenty day 
cumulative criterion was the two minute time interval. Coefficients 
resulting from these comparisons are shown in Table 3. Although the 
pattern of acceptable coefficients at . 85 or above was almost the same as 
the thirty second interval, there was one striking difference noted. It 
was found that after only three days of recording the activities of six 
pupils, an acceptable pattern of indices resulted. After this point on the 
matrix, there were no indices lower than .82 either across or below, and 
there were no three consecutive indices below . 85 except for the activities 
recorded for twenty-seven and twenty-eight pupils. This high level 
consistency did not occur with so few days or pupils with either the thirty 
second or the one minute time interval. Also, the two minute time interval 
indicated acceptable consistency in eleven days when one pupil was observed 
each day; thio was three days sooner than the every one minute interval. 
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But, as with the thirty second and the one minute interval, no consistent 
acceptable level of coefficients was found for thirty-three pupils observed 
for only one day. After the examination of the data yielded with the every 
two minute recording interval, it was concluded that for a thirty minute 
observation period the two minute time interval was efficient as the every 
thirty-second interval. 

Continuing the examination of varying the length of the time 
interval the next observation interval explored was the every four minute 
interval. Scanning the indices shown in Table 4 indicated that there were 
very few indices at .85 or above for the first seven days of observation. 
Although it was noted that when the activities of four pupils were cumulated 
for eight days, the pattern of indices was consistent and acceptable thereafter. 
Furthermore, it was noted that on the eighth day the indices for four through 
thirty-three pupils tended to rise to .90 and above and remained at this 
very high level for the remainder of the matrix. The general conclusion 
drawn from the exploration of this interval was that a few more pupils and 
days were necessary to produce a consistent acceptable level of agr jement 
than were necessary with the smaller time intervals. 

The indices that resulted from comparing the s?x minute time 
interval and the criterion measure are shown in Table 5. Investigation of 
this matrix showed that the upper triangle had few indices at . 85 or above 
and that over one-half of the indices in the entire matrix were below .85. 
Moreover, when the activities of only one pupil were recorded each day, 
an acceptable level of agreement was not reached with the criterion measure 
even on the twentieth day. Furthermore, during the first seven days of 
recording there were not satisfactory indices computed. Actually, nine 
days of recording the activities of nineteen pupils were necessary to attain 
and then maintain an acceptable level of agreement. 

The eight minute time interval was the next interval to be 
examined. As seen in Table 6, this interval did produce acceptable 
indices a few days earlier than the six minute interval. But, the eight 
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minute interval required the cumulative observations of three pupils 
for sixteen days to yield a consistent acceptable level, whereas the 
six minute interval required only two pupils to produce the same results. 
Further exploration of the time interval led to the examination 
of the every ten minute interval. Table 7 shows the matrix that resulted 
from comparing the measurements of this interval with the twenty day 
every thirty seconds cumulative criterion. A few more days and pupils 
were required to reach consistent acceptable levels than with the eight 
minute time interval even though approximately one -half of the total 
coefficients computed were . 85 or above. 

The resulting indices of agreement of the twelve minute interval 
are shown in Table 8. These data indicated that at least six pupils' 
activities had to be cumulated even on the twentieth day of observation to 
produce a consistent .85 or above, Also,. at least twenty-nine pupils had 
to be observed for eight days to yield an acceptable level with this interval. 
Less than one-half of the coefficients computed in the entire matrix were 
at . 85 or above. 

The last interval to be explored was the fourteen minute time 
interval. These coefficients are shown in Table 9* Even though more 
than one-half of the indices in the matrix were either . 85 or above, at 
least eight days of observing the activities of ten pupils were necessary 
to produce a consistent acceptable level. It was noted, however, that the 
ten and twelve minute intervals did not show this early acceptance level 
with only ten pupils. 

From this investigation of the time interval, it was concluded 
that even though the every thirty-second time interval produced the most 
consistent acceptable level of agreement, the every one minute and the 
every v/o minute time intervals were satisfactory and could be utilized 
efficiently. By increasing the number of days of observation and the 
number of pupils being observed, the four minute inte r val could also be 
employed. The six, eight, ten, twelve, and fourteen minute time intervals 
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arc not recommended because of the large number of pupils and 
observation periods required. However, if the physical arrangement 
of the school permitted the observer to move freely from one classroom 
to another during the same class period, it laight be possible to efficiently 
observe a few pupils in several classrooms during the same class period 
by using the larger time intervals between recordings, and by adding a 
few more days of observation. For example, three pupils observed in 
each of several classrooms every six, eight, or ten minutes for sixteen 
days may produce consistent acceptable results. 

This paper has thus far discussed three of the four variables 
considered in this study. While the observalion period was held constant 
at thirty minutes, the variable time interval, number of days, and number 
cf pupils were manipulated. Now the fourth variable, the observation 
period, will be examined. The time interval was held constant at every 
thirty-seconds for this portion of the study while the observation period 
was varied at 5, 10, 15, and minutes. Also, the thirty minute observa- 
tion period, which has already been discusse d, can be added to this series. 

The matrix that resulted from comparing the five minute obser- 
vation period for one through thirty-three pupils and the twenty day cumulative 
criterion is shown in Table 10, It was noted that although there were high 
levels of agreement reached at various points on the matrix, they were not 
always followed by other high indices. Fifteen days of observing one pupil 
each day were required to attain an acceptable level. Moreover, even though 
the matrix shows that on the first day of observation high levels were reached 
with as few as twenty-one pupils, an acceptable level of acceptance was not 
maintained with additional days. 

Although the matrix computed for the five minute observation 
period appeared to indicate that a five minute observation period was 
sufficient, there was doubt concerning its acceptance because of several 
inconsistencies occurring throughout the matrix. This doub*; was 
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substantiated when the ten minute observation period, shown in Table 11, 
was explored. At several points on the matrix the indices were lower 
for the ten minute observation period than for the five minute period. 

Since consistent acceptable levels were not maintained for the ten minute 
period, neither the five nor the ten znirute observation period could be 
recommended unless a sufficient number of pupils and days were added to 
overcome the inconsistencies that occur. In this study the ten minute 
period required that at least five pupils be observed for twenty days to 
attain and maintain a consistent acceptabel level. However, observing 
twenty-one pupils for two days was sufficient to attain an acceptable level 
and no three consecutive indices fell below . 85 after this point on the matrix. 

The next observation period that was investigated for twenty 
class periods was hie fifte en minute observation period. The coefficients 
for this observation period are shown in Table 12. Investigation showed 
that the general level of agreement of corresponding indices increased 
over those computed for the five and ten minute periods. Following the 
general rule of acceptance, it was found that seventeen days of observation 
were required when only one pupiFs activities were recorded daily while 
only twelve days were needed for two pupils and nine days for three pupils. 
Also, if twelve pupils were observed foi approximately three days, 
fifteen minutes was a sufficient observation period. 

Twenty minutes was the next observation period examined. 

Table 13 shows the coefficients that resulted from comparing this 
observation period with the twenty day criterion. When one pupil was 
selected randomly each clay, nineteen days were required to attain a 
consistent acceptable level of .85 or above. But when observational 
measurements were made on fourteen pupils, acceptable coefficients 
wore yielded after only two days of observance. Comparing these 
coefficients with those obtained for the thirty minute observation period 
(Table 1) revealed that the twenty minute observation period was as 
acceptable as the thirty minute observation period. Only a few more 
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indices fell below the . 85 acceptance level than with the thirty minute 
observation period. Furthermore, approximately two -thirds of the 
coefficients for the twenty minute observation period were « 90 or above. 
However, the column for thirty-three pupils does show slightly lower 
levels of agreement than those computed for the thirty minute period 
but both observation periods showed consistent acceptable levels were 
maintained for thirty-three pupils after the second day's activities were 
added. It was concluded, therefore, that twenty minutes was a sufficient 
observation period for the every thirty-second time interval. 

General Conclusions 

Several general conclusions wore drawn from this study* 

(1) It was found that when the time interval between recording 
observations and the length of the observation period were held constant, 
the number of pupils necessary to observe to reach an acceptable level of 
agreement was inversely related to the number of days of observation. 

(2) The one minute and the two minute time intervals were as 
efficient as the every thirty second recording interval when the observation 
period was thirty minutes in length. 

(3) With a few more additional pupils and days, the four minute 
time interval proved to be as satisfactory as the smaller time intervals if 
the observation period was thirty minutes in length. 

(4) The six, eight, ten, twelve, and fourteen minute time 
intervals were not adequate. Several additional days or pupils had to be 
added in order to overcome the lack of a sufficient number of observations 
needed to produce a consistent and acceptable level of agreement. There- 
fore, these time intervals are not generally recommended. 

(5) When the time interval between recording observations was 
thirty seconds, the twenty minute observation period was as efficient as 
the thirty minute period. 

(6) When a few additional days or pupils were added, the fifteen 
runute observation period and the every thirty seconds time interval 



were as satisfactory as the thirty minute observation period and the 
every thirty seconds time interval. 

(7) The five and ten minute observation periods were not 
efficient observation periods with the every thirty seconds interval. 
Therefore, these observation periods are not generally recommended. 

In summary, the general conclusion reached is that it is not 
necessary to record the behavior of every pupil in order to describe the 
activities of an entire class. Efficient methods of observation are 
possible through various sampling plans and observational methods. 

Application 

This study has identified several efficient sampling designs 
for studying pupil classroom activities. It has also shown that it is 
not always necessary to record the activities of the entire class in order 
to assess total class behavior# The amount of time and money spent on 
observational measurements, therefore, can be reduced if efficient 
designs are employed. Adaptations of the designs presented in this study 
could be utilized easily in many classroom studies which compare pupil 
classroom activities either between schools or within schools and either 
across grade levels or within the same grade level. These designs might 
also be used to test innovative classroom procedures or to compare new 
procedures to existing classroom methods in order to determine the extent 
to which desired'pupil activity changes are occurring. In addition, this 
research could be utilized to direct further methodological studies of the 
same type using either the same or different variables. 
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♦Underlined values indicate agreement level of . 85 or greater. 
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